{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/recommendation/article-recommender/article_recommendations.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/recommendation/article-recommender/article_recommendations.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fb53FkdrRB-K"
      },
      "source": [
        "# Personalized Article Recommendation Engine (Example)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRoiji_IRB-O"
      },
      "source": [
        "This notebook demonstrates how to use Pinecone's similarity search to create a simple personalized article or content recommender."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E3wl_OoRRB-P"
      },
      "source": [
        "The goal is to create a recommendation engine that retrieves the best article recommendations for each user. When making recommendations with content-based filtering, we evaluate the user\u2019s past behavior and the content items themselves. So in this example, users will be recommended articles that are similar to those they've already read.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Of_-qU8RB-P"
      },
      "source": [
        "## Install and Import Python Packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UOjd2xMJRB-P",
        "outputId": "7691f20f-ef2b-4c87-98e8-634cb68cd9d3"
      },
      "outputs": [],
      "source": [
        "!pip install --quiet \\\n",
        "    matplotlib==3.10.1 \\\n",
        "    pandas==2.2.3 \\\n",
        "    pinecone==6.0.2 \\\n",
        "    sentence-transformers==4.0.1 \\\n",
        "    tf-keras==2.19.0 \\\n",
        "    torch==2.6.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "PodOM8HCDbrG"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import re\n",
        "import matplotlib.pyplot as plt\n",
        "from statistics import mean\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRgkJi7tRB-T"
      },
      "source": [
        "In the following sections, we will use Pinecone to easily build an article recommendation engine. Pinecone will be responsible for storing embeddings for articles, maintaining a live index of those vectors, and returning recommended articles on-demand. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0zk26pXDbrH"
      },
      "source": [
        "## Pinecone Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/opt/conda/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "from pinecone import Pinecone\n",
        "\n",
        "# Initialize connection to Pinecone (obtain API key at app.pinecone.io)\n",
        "api_key = os.environ.get(\"PINECONE_API_KEY\") or \"PINECONE_API_KEY\"\n",
        "\n",
        "# Configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create the index:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "index_name = \"articles-recommendation\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{\n",
              "    \"name\": \"articles-recommendation\",\n",
              "    \"metric\": \"cosine\",\n",
              "    \"host\": \"articles-recommendation-dojoi3u.svc.aped-4627-b74a.pinecone.io\",\n",
              "    \"spec\": {\n",
              "        \"serverless\": {\n",
              "            \"cloud\": \"aws\",\n",
              "            \"region\": \"us-east-1\"\n",
              "        }\n",
              "    },\n",
              "    \"status\": {\n",
              "        \"ready\": true,\n",
              "        \"state\": \"Ready\"\n",
              "    },\n",
              "    \"vector_type\": \"dense\",\n",
              "    \"dimension\": 300,\n",
              "    \"deletion_protection\": \"disabled\",\n",
              "    \"tags\": null\n",
              "}"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "# Check if index already exists (it shouldn't if this is first time)\n",
        "if not pc.has_index(name=index_name):\n",
        "    # If does not exist, create index\n",
        "    pc.create_index(\n",
        "        index_name,\n",
        "        dimension=300,\n",
        "        metric=\"cosine\",\n",
        "        spec=ServerlessSpec(cloud=\"aws\", region=\"us-east-1\"),\n",
        "    )\n",
        "\n",
        "index_config = pc.describe_index(name=index_name)\n",
        "index_config"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 300,\n",
              " 'index_fullness': 0.0,\n",
              " 'metric': 'cosine',\n",
              " 'namespaces': {'articles': {'vector_count': 189760}},\n",
              " 'total_vector_count': 189760,\n",
              " 'vector_type': 'dense'}"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Instantiate the Index client\n",
        "index = pc.Index(host=index_config.host)\n",
        "\n",
        "# View index stats\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "05yNGSguRB-V"
      },
      "source": [
        "## Upload Articles\n",
        "Next, we will prepare data for the Pinecone vector index, and insert it in batches."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RfKRg-IqRB-R"
      },
      "source": [
        "This demonstration is built using the  [All the News 2 dataset](https://components.one/datasets/all-the-news-2-news-articles-dataset/) which is available for non-commercial research use. The full dataset contains 2.7 million news articles and essays from 27 American publications, but for expediency we will be working with a truncated set of 200k entries. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gu4sAgkfRB-R"
      },
      "source": [
        "First, let's define some functions to help with downloading the data we need."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J5xQaXWmRB-R",
        "outputId": "fb0ff131-d855-4cf5-a0ec-133f1315e553"
      },
      "outputs": [],
      "source": [
        "import urllib.request\n",
        "import zipfile\n",
        "\n",
        "\n",
        "def progress_bar(block_num, block_size, total_size):\n",
        "    downloaded = block_num * block_size\n",
        "    if total_size > 0:\n",
        "        percent = downloaded * 100 // total_size\n",
        "        print(\n",
        "            f\"\\rDownloading: [{'#'*percent}{' '*(100-percent)}] {percent:.2f}% \", end=\"\"\n",
        "        )\n",
        "    else:\n",
        "        print(f\"\\rDownloading: {downloaded} bytes\", end=\"\")\n",
        "\n",
        "\n",
        "def download(url, filename):\n",
        "    # Download the file\n",
        "    print(\"Downloading dataset...\")\n",
        "    urllib.request.urlretrieve(url, filename, reporthook=progress_bar)\n",
        "\n",
        "\n",
        "def unzip(zipfilename):\n",
        "    # Unzip the file\n",
        "    print(\"Extracting dataset...\")\n",
        "    with zipfile.ZipFile(zipfilename, \"r\") as zip_ref:\n",
        "        zip_ref.extractall(\".\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "\n",
        "def download_full_dataset():\n",
        "    \"\"\"The full dataset has 2.7M articles and may cause your notebook environment to crash\"\"\"\n",
        "    zipfile = \"all-the-news-2-1.zip\"\n",
        "    if not os.path.exists(zipfile):\n",
        "        url = \"https://storage.googleapis.com/examples-repo-public/article-recommender/all-the-news-2-1.zip\"\n",
        "        download(url, zipfile)\n",
        "        unzip(zipfile)\n",
        "\n",
        "\n",
        "def download_truncated_dataset():\n",
        "    \"\"\"This function downloads a sample of 200k articles from the full 2.7M article dataset\"\"\"\n",
        "    zip_file = \"all-the-news-200k.zip\"\n",
        "    if not os.path.exists(zip_file):\n",
        "        url = \"https://storage.googleapis.com/examples-repo-public/article-recommender/all-the-news-200k.zip\"\n",
        "        download(url, zip_file)\n",
        "        unzip(zip_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's download the dataset. This will take a few minutes to complete. \n",
        "\n",
        "If you prefer to work with the full data, you can call `download_full_dataset()` instead of `download_truncated_dataset()` below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 3 \u03bcs, sys: 1 \u03bcs, total: 4 \u03bcs\n",
            "Wall time: 10.5 \u03bcs\n"
          ]
        }
      ],
      "source": [
        "%time\n",
        "\n",
        "# download_full_dataset() # Use this instead for full 2.7M record dataset\n",
        "download_truncated_dataset()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p-jcFPPaRB-S"
      },
      "source": [
        "### Create Vector Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hytn3DeYRB-S"
      },
      "source": [
        "The model used in this example is the [Average Word Embeddings Models](https://www.sbert.net/docs/sentence_transformer/pretrained_models.html). This model allows us to create vector embeddings for each article, using the content and title of each."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273,
          "referenced_widgets": [
            "6fbe62563e734d0c9044071436a877a7",
            "77e44f6033ad4b3aa33fb962aa7cb204",
            "582d963297b24592bd8fd06f04431af1",
            "36cce955f93e4b33b425ed41e0b4ba30",
            "a3584eefc06e4d5ba1a857ca6c720396",
            "88cc837dea3e41e38ec169efaeacd76d",
            "e9066d377b1843f9bbf6c4e1d4e55dba",
            "9d20563d50644abf8a958b9bb77b2d0f",
            "2e3db13994854735b4d4f75ae48ccfc6",
            "60c15fff6daf48a493f2fb7a24264f60",
            "9662f75bbe0441cbbb3d4d53e79862b7",
            "2011fef509144548b1663b73b2466ab3",
            "318827a30c8c4bf095b0bbc834815a56",
            "9810b57f422c4cd79e5cb6e79bd6619f",
            "a8032f79f295466bbf016309e731f659",
            "383c8e8faff648acb828eee26128b97c",
            "0c66e0deab1046a883d55afd2a30c655",
            "4ac18b0659fc4707abfa4451bcd88ac2",
            "bb3971c788e14e55aa6dbf21ed7d5413",
            "03d4f3f260da460293bcf0549a2c19ad",
            "8b2c3e18800045c19e2d3c24b574e7eb",
            "53a6be8d51c942e09605327ba1a1adad",
            "2c69b8bc324e4bda98ac7ecbb24808da",
            "6e77800741174a9bb95c5670efaa1a57",
            "1fa893ce4cf8425db6e12bb9fbd67f36",
            "83e02c01be5c460d8e539e4154d5f25c",
            "05932d019bfa4cd3a3f49ec5446c7896",
            "4da918342e8740e6b8932ed1c3b461bd",
            "0e247ffc057e4818b7e712def59f4b93",
            "028c233ea83747328dcaf0f87480ea43",
            "fbe75f11f8874b74ada3a51b2e2d16f1",
            "b65a89f79b4947bcbbe841a46b2ddf69",
            "c4e236245fd04112ad783aedd621f658",
            "8619c53b7bc14127a9845e01b4632db1",
            "773081b39ede4124a384316fd7f6f90d",
            "705fbd6cec0d48259724882007677083",
            "67ec242030034acbad9b195531030b68",
            "3ad20c39761e46dcbc7abd21a3aed02d",
            "346ec4000e884dcc87b6e1b3ad5bdbf2",
            "3745f975cc684b0483a2c7fbabdb6997",
            "60d34cb257954b1abdcae3727b810456",
            "3ace2e07311b4877b974c7b7ffd2af3a",
            "a6d2b8f0a7b2438d9cdfab3b01bc5de0",
            "8b0d9bf73a60478ba9ba3917a035f7f2",
            "575689d5031b40ffabcdf3439b260f42",
            "ed42d11fbe9a49bbb1fd235ee79ff754",
            "631b429f21d041a2b49fb6351596d97d",
            "8a959a5639c342d2b6794d94044bef42",
            "54ce280c11174ef6a8d45e4f16c24ed8",
            "3069962424a743c79815474986638caf",
            "23a0d9d55d2a40cc9dacf7c716a8028e",
            "69919865e60a4c5d9f5cb9ca46b61c13",
            "5e1d703809784a50af2f044d9d32ee24",
            "896961dc72c64256b625fbfb474cadf5",
            "3fa8a354f3204a4aa672183648a24e20",
            "482140bf674c4e9896bb664b5210e0ed",
            "9847e6b5fea94016ac49f887f3d61662",
            "b2185fb252474f3fa2cc86ce4e828cce",
            "6b42f6dc440643338ef5424a43f7a7bc",
            "ee1c910059a64f5aa832354d691db6ad",
            "bb7d1df01bda46ef8b202f10a6187521",
            "5516f07b182d481686d099e007a3d34b",
            "80b927762023402cbd1c1a1ab9dc4db3",
            "4b4c04a5183242bbb9a9cfd9334eb09c",
            "0e325ca865414b6bb7a6ce06930d1581",
            "851c1c4fa33946e58bc8b98b92e27b8c",
            "4d1f238c091e4b89abd03c0b203cdf89",
            "176744bb5eaf4284bdb3ed77487f8aec",
            "77231d482ac7477193f82e8e4fa55cb0",
            "2c93602907ae4fe9af718c1184fe9cda",
            "6a6b6d6b5a6c4c5c84a23deef6d4c273",
            "7bb0f1d9694646189c4b67cfed516c87",
            "59d3321e250249c3b30565953667adf3",
            "3916a460acd04feeaecc7b4e89b06549",
            "febace71f0a44cc0b329e2362d5d7e59",
            "4ed58e59df354489bf40df70abb7a995",
            "dd716da7710f4144ba258114786bd145",
            "421675451d674bacbe6c35a2895a9ee5",
            "aa317bc58b2847729846140b0a1fc6e3",
            "cffbbfec40174b9c97f66345b32074f1",
            "96492621d08d431f9be8c6c9f19ad2ef",
            "c8b7975d8eab4391a292bb817dee4176",
            "a5a8a96435c04c638194ed53c7c3d0a9",
            "3a04aa5acd044e349a9635ef7533d87e",
            "7baaca7f6eb44a2a8c734c0dd05a74b4",
            "c051c6d3d07c44bc82a8d507a4caaa27",
            "37f09f44c57249c6b77f201a8e14122a",
            "451bbd2981dd4131a2da1d6af7c9261a"
          ]
        },
        "id": "VxzOvUY3DbrO",
        "outputId": "e7f2cc13-8f98-499f-b804-06c0527e5d04"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "# set device to GPU if available\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model = SentenceTransformer(\"average_word_embeddings_komninos\", device=device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "F0dUR-mMDbrP"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 100  # batch size for upserting"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "# CSV_FILENAME = \"all-the-news-2-1.csv\" # use this if you opted to download the full dataset\n",
        "CSV_FILENAME = \"all-the-news-200k.csv\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GYouYV1DDbrP"
      },
      "source": [
        "Let's prepare data for upload."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aQwP2ufHRB-W",
        "outputId": "f8127abe-d003-492f-f857-fb94244dd20c",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "import hashlib\n",
        "from tqdm.auto import tqdm\n",
        "from pprint import pprint\n",
        "\n",
        "\n",
        "def hash_title(title: str) -> str:\n",
        "    \"\"\"Generate a SHA-256 hash of the title.\"\"\"\n",
        "    return hashlib.sha256(title.encode(\"utf-8\")).hexdigest()\n",
        "\n",
        "\n",
        "def metadata(row) -> dict:\n",
        "    meta = {\n",
        "        \"title\": row.title,\n",
        "        \"article\": row.article[:400],\n",
        "        \"section\": row.section,\n",
        "        \"publication\": row.publication,\n",
        "    }\n",
        "    # pprint(meta)\n",
        "    return meta\n",
        "\n",
        "\n",
        "def create_embeddings(data) -> pd.DataFrame:\n",
        "    \"Preprocesses data and prepares it for upsert.\"\n",
        "\n",
        "    # Drop records where 'title' or 'article' are empty string\n",
        "    data[\"article\"] = data[\"article\"].fillna(\"\")\n",
        "    data[\"title\"] = data[\"title\"].fillna(\"\")\n",
        "    data[\"publication\"] = data[\"publication\"].fillna(\"\")\n",
        "    data[\"section\"] = data[\"section\"].fillna(\"\")\n",
        "\n",
        "    data = data[data[\"title\"].str.strip() != \"\"].copy()\n",
        "    data = data[data[\"article\"].str.strip() != \"\"].copy()\n",
        "    data = data.reset_index(drop=True)\n",
        "\n",
        "    # Add an id column\n",
        "    data[\"id\"] = data[\"title\"].apply(hash_title)\n",
        "\n",
        "    # Truncate article text\n",
        "    data[\"article\"] = data.article.apply(\n",
        "        lambda x: \" \".join(re.split(r\"(?<=[.:;])\\s\", x)[:4])\n",
        "    )\n",
        "    data[\"title_article\"] = data[\"title\"] + data[\"article\"]\n",
        "\n",
        "    # Create a vector embedding based on title and article columns\n",
        "    encoded_articles = model.encode(data[\"title_article\"])\n",
        "    data[\"article_vector\"] = pd.Series(encoded_articles.tolist())\n",
        "\n",
        "    return data\n",
        "\n",
        "\n",
        "def upload_items(data):\n",
        "    \"\"\"Uploads data to the Pinecone index.\"\"\"\n",
        "\n",
        "    # create a list of items for upload\n",
        "    items_to_upload = [\n",
        "        (str(row.id), row.article_vector, metadata(row)) for i, row in data.iterrows()\n",
        "    ]\n",
        "\n",
        "    # upsert\n",
        "    index.upsert(vectors=items_to_upload, namespace=\"articles\", async_req=True)\n",
        "\n",
        "\n",
        "def process_file(filename: str) -> pd.DataFrame:\n",
        "    \"Reads csv files in chunks, prepares and uploads data.\"\n",
        "\n",
        "    with tqdm(total=200000, desc=\"records processed\") as pbar:\n",
        "        for chunk in pd.read_csv(filename, chunksize=BATCH_SIZE):\n",
        "            data = create_embeddings(chunk)\n",
        "            upload_items(data)\n",
        "            pbar.update(BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "records processed: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 200000/200000 [02:41<00:00, 1239.07it/s]\n"
          ]
        }
      ],
      "source": [
        "process_file(filename=CSV_FILENAME)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IP75eE73RB-W",
        "outputId": "726c8edf-9680-412d-c489-ab9422a6ff4b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 300,\n",
              " 'index_fullness': 0.0,\n",
              " 'metric': 'cosine',\n",
              " 'namespaces': {'articles': {'vector_count': 189760}},\n",
              " 'total_vector_count': 189760,\n",
              " 'vector_type': 'dense'}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Print index statistics\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hzFewYUHRB-W"
      },
      "source": [
        "## Query the Pinecone Index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FMUxnkYHDbrT"
      },
      "source": [
        "We will query the index to get recommendations for different types of users. For each user we will select 10 articles matching keywords related to their areas of interest and use that to represent their reading history. Then we will average the embeddings for each of these articles to get a query vector we can use to find other similar articles to recommend.\n",
        "\n",
        "To demonstrate, we will query Pinecone to get recommendations for users matching these descriptions:\n",
        "* User who likes to read Sport News\n",
        "* User who likes to read Entertainment News\n",
        "* User who likes to read Business News"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "YKkpgH7CRB-X"
      },
      "outputs": [],
      "source": [
        "def display_recommendations(recommendations: pd.DataFrame):\n",
        "    print(\"Recommended Articles\")\n",
        "    for i, row in recommendations.iterrows():\n",
        "        print(f\"  - {row.title}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "\n",
        "def get_simulated_reading_history(filter_chunk_fn):\n",
        "    filtered_rows = []  # To store matching rows\n",
        "    target_count = 10\n",
        "\n",
        "    # Read the CSV in chunks. This adds complexity, but allows us\n",
        "    # to work with large CSV files without loading everything at\n",
        "    # once.\n",
        "    for chunk in pd.read_csv(CSV_FILENAME, chunksize=10000):\n",
        "        # Filter the chunk using the function above\n",
        "        filtered_chunk = filter_chunk_fn(chunk)\n",
        "\n",
        "        if not filtered_chunk.empty:\n",
        "            filtered_rows.append(filtered_chunk)\n",
        "\n",
        "        # If we've collected enough rows, break out of the loop early\n",
        "        if sum(len(df) for df in filtered_rows) >= target_count:\n",
        "            break\n",
        "\n",
        "    # Concatenate the filtered results and select the first 10 rows\n",
        "    if filtered_rows:\n",
        "        result = pd.concat(filtered_rows).iloc[:target_count]\n",
        "    else:\n",
        "        result = pd.DataFrame()\n",
        "\n",
        "    return result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [],
      "source": [
        "def vector_average(articles_read):\n",
        "    a = articles_read[\"article_vector\"]\n",
        "    user_vector = [*map(mean, zip(*a))]\n",
        "    return user_vector\n",
        "\n",
        "\n",
        "def recommendations(articles_read):\n",
        "    query_vector = vector_average(articles_read)\n",
        "    res = index.query(\n",
        "        vector=query_vector, top_k=20, namespace=\"articles\", include_metadata=True\n",
        "    )\n",
        "    df = pd.DataFrame([m.metadata for m in res.matches])\n",
        "\n",
        "    # Remove articles the user has already read since it doesn't make sense to recommend these\n",
        "    df = df[~df[\"title\"].isin(articles_read[\"title\"])]\n",
        "\n",
        "    return df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3EGTDEZ5DbrV"
      },
      "source": [
        "Now that we have defined these utility functions, let's see some examples. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Recommendations for tennis fan\n",
        "\n",
        "Let's choose 10 articles from our dataset about Tennis to use as a user's reading history. Each piece of content has an embedding associated with it. We can average these embeddings and use that average vector to find other similar content in the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Articles user has read:\n",
            "  - Players and fans swelter in Melbourne, brace for worse on Friday\n",
            "  - Australia's Hewitt says ATP Cup success put ITF under pressure\n",
            "  - Tennis: Spanish player banned for eight months for betting offences\n",
            "  - Tennis: Cecchinato in dreamland two years on from match-fixing ban\n",
            "  - TIU bans Australian Lindahl for 2013 match-fixing scandal\n",
            "  - U.S. draw Latvia in 2020 Fed Cup qualifiers\n",
            "  - Bernard Tomic's Only Sin Was Being Honest About His Job\n",
            "  - Tennis: U.S. Open to top $50 million in prize money\n",
            "  - Tennis: New French Open court unveiled as prize money increases\n",
            "  - Ukrainian twins banned for life for match-fixing: TIU\n"
          ]
        }
      ],
      "source": [
        "def sports_filter(chunk):\n",
        "    return chunk[\n",
        "        ((chunk[\"section\"] == \"Sports News\") | (chunk[\"section\"] == \"Sports\"))\n",
        "        & (chunk[\"article\"].str.contains(\"Tennis\", na=False))\n",
        "    ]\n",
        "\n",
        "\n",
        "sports_articles_read = get_simulated_reading_history(sports_filter)\n",
        "sports_articles_read = create_embeddings(sports_articles_read)\n",
        "\n",
        "print(\"Articles user has read:\")\n",
        "for i, row in sports_articles_read.iterrows():\n",
        "    print(f\"  - {row.title}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Recommended Articles\n",
            "  - No French Open wild card for snubbed Sharapova\n",
            "  - Sharapova handed wild card for Rogers Cup in Toronto\n",
            "  - Wilander criticizes Davis Cup revamp as 'last' final looms\n",
            "  - Roger Federer Survives a Scare, and Rafael Nadal Overcomes a Din\n",
            "  - Novak Djokovic\u2019s Invincibility Takes Another Hit, Opening a Door for Others\n",
            "  - Golf: Confident Oosthuizen puts on his game face for Presidents Cup\n",
            "  - Serena 'ready to go' says Fed Cup captain Rinaldi\n",
            "  - French Federation extends Paire suspension\n",
            "  - Tennis: Wawrinka feared for career during long layoff\n",
            "  - Fleetwood prepares for special homecoming at British Open\n",
            "  - Basketball : It looks like the U.S. and then all the rest\n",
            "  - Russia back in the fray despite build-up distractions\n",
            "  - Kvitova withdraws from Birmingham Classic\n",
            "  - Everything to Know About U.S. Open Champion Naomi Osaka\n",
            "  - Real Madrid could break spending record to rescue Neymar from PSG\n",
            "  - In Women\u2019s Tennis, Finesse Can Fight Power\n"
          ]
        }
      ],
      "source": [
        "sports_recs = recommendations(sports_articles_read)\n",
        "display_recommendations(sports_recs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQetrCScRB-X"
      },
      "source": [
        "### Recommendations for a video game fan"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "4lnd4lI7RB-Y",
        "outputId": "b58f67b4-6cb2-4be2-de1c-d89f114a5c58"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Articles user has read:\n",
            "  - How a Horror Game About Death Expanded Its Audience By Letting Them Live\n",
            "  - \u2018Warframe\u2019 Hacked, Details on 775,000 Players Traded\n",
            "  - Mike Patton\u2019s the Best Rock Star to Have Ever Rolled Into Video Games\n",
            "  - We Asked Nintendo, Microsoft, and 12 Other Devs How They Deal With Crunch\n",
            "  - Microsoft, Nintendo and Sony's Next Gen Consoles According to E3\n",
            "  - 'PUBG' Has an NFL Red Zone Channel Skipping to the Best Part of Every Match\n",
            "  - Reexamining \u2018Silent Hill 3,\u2019 Gaming\u2019s Most Unfairly Overlooked Sequel\n",
            "  - The Best Work Motherboard Published in 2017\n",
            "  - 'Forza Horizon 4' Is a Living Impressionist Landscape\n",
            "  - Why We\u2019re Still Addicted To \u2018Rocket League,' One Year Later\n"
          ]
        }
      ],
      "source": [
        "def xbox_filter(chunk):\n",
        "    return chunk[\n",
        "        (\n",
        "            (chunk[\"section\"] == \"Entertainment\")\n",
        "            | (chunk[\"section\"] == \"Games\")\n",
        "            | (chunk[\"section\"] == \"Tech by VICE\")\n",
        "        )\n",
        "        & (chunk[\"article\"].str.contains(\"Xbox\"))\n",
        "    ]\n",
        "\n",
        "\n",
        "xbox_articles_read = get_simulated_reading_history(xbox_filter)\n",
        "xbox_articles_read = create_embeddings(xbox_articles_read)\n",
        "\n",
        "print(\"Articles user has read:\")\n",
        "for i, row in xbox_articles_read.iterrows():\n",
        "    print(f\"  - {row.title}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Recommended Articles\n",
            "  - Pre-Ordering \u2018Battlefield 1\u2019 for $150 Is a Bad Idea But I Did It Anyway\n",
            "  - The best video games of 2019, from 'Apex Legends' to 'Zelda' \u2014 LIST\n",
            "  - First Click: How much is an hour of entertainment worth? | The Verge\n",
            "  - Fortnite is finally coming to Android this summer \u2013 TechCrunch\n",
            "  - We Need More Games That Take Chances With Major Franchises\n",
            "  - PS4\u2019s exclusive zombie game Days Gone deserves a chance to be itself\n",
            "  - Fortnite season 5: What we know so far\n",
            "  - Dangerous, explosive hoverboards are still lurking, and here's another recall to remind you of it\n",
            "  - 'The Last of Us Part II' delayed: New release date, trailer, gameplay\n",
            "  - How to Explain Speedrunning to Your Parents\n",
            "  - Watch the Xbox Scorpio unveiling live, right here\n",
            "  - Looking Back on 2015's Biggest Tech News and Ahead to CES 2016\n",
            "  - Nintendo is releasing a massive Splatoon 2 update in time for the holidays\n",
            "  - 'Super Mario Maker 2' Gives You More Tools to Create Chaos\n",
            "  - We Talk About Game Reviews And Loving 'Titanfall 2' On Waypoint\u2019s New Podcast\n",
            "  - No Man\u2019s Sky\u2019s upcoming update paves the way for building bases \u2013 TechCrunch\n",
            "  - 'Pok\u00e9mon Go' gets sponsored locations in advertising push\n",
            "  - Meet the new Instagram influencers: houses with cult followings\n",
            "  - The Online Community That's Turning Old Video Games Into an Archaeological Dig\n",
            "  - Review: \u2018Taken,\u2019 a Prequel, Offers the Slaughter Before the Slaughter\n"
          ]
        }
      ],
      "source": [
        "xbox_recs = recommendations(xbox_articles_read)\n",
        "display_recommendations(xbox_recs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8Px1ifjRB-Y"
      },
      "source": [
        "### Recommendations for Business User"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "jC_OrP41RB-Y",
        "outputId": "940e5396-53b1-4f47-8385-581aff72fccc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Articles user has read:\n",
            "  - Wall St. falls as investors eye a united hawkish Fed\n",
            "  - Why Cloudflare Let an Extremist Stronghold Burn\n",
            "  - China Stocks Plunge as Coronavirus Fears Grow\n",
            "  - Morgan Stanley elbows out rivals for plum role in $1.5 billion IPO relaunch: sources\n",
            "  - New York City Pension Fund to Divest Itself of Gun Retailer Stock\n",
            "  - Exxon quarterly profit falls 5.2% on weak refining, chemical margins\n",
            "  - PepsiCo's mini-sized sodas boost quarterly results\n",
            "  - Wall Street extends rally on U.S.-China trade optimism\n",
            "  - DealBook Briefing: Could \u2018Down Round\u2019 I.P.O.s Hit the Tech Unicorns?\n",
            "  - Caesars must face $11 billion in lawsuits: U.S. judge\n"
          ]
        }
      ],
      "source": [
        "def business_filter(chunk):\n",
        "    return chunk[\n",
        "        ((chunk[\"section\"] == \"Business News\") | (chunk[\"section\"] == \"business\"))\n",
        "        & (chunk[\"article\"].str.contains(\"Wall Street\"))\n",
        "    ]\n",
        "\n",
        "\n",
        "business_articles_read = get_simulated_reading_history(business_filter)\n",
        "business_articles_read = create_embeddings(business_articles_read)\n",
        "\n",
        "print(\"Articles user has read:\")\n",
        "for i, row in business_articles_read.iterrows():\n",
        "    print(f\"  - {row.title}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Recommended Articles\n",
            "  - GLOBAL MARKETS-Stocks tumble as coronavirus cases rise outside China\n",
            "  - GLOBAL MARKETS-Asia shares reassured by Powell pick, pause for US jobs test\n",
            "  - GLOBAL MARKETS-Asia shares dragged lower by Japan, dollar still groggy\n",
            "  - GLOBAL MARKETS-Carmakers drive Europe higher, Johnson batters sterling\n",
            "  - Unilever touts new products, structure to fight rivals\n",
            "  - GLOBAL MARKETS-Asian shares retreat from highs, markets take Trump impeachment in stride\n",
            "  - UPDATE 1-South Africa's rand lifted by EM demand, manufacturing surprise\n",
            "  - UPDATE 1-European shares edge higher as miners and banks gain\n",
            "  - European shares edge higher on trade hopes, Imperial Brands slides\n",
            "  - UPDATE 3-Australia's Blackmores changes tack in China as profit falls\n",
            "  - UPDATE 3-Slump in toys and gaming sales spoils Sainsbury's Christmas\n",
            "  - CORRECTED-European stocks flat ahead of U.S. payrolls\n",
            "  - EMERGING MARKETS-Latam FX, stocks down; Argentine stocks surge ahead of primary vote\n",
            "  - UPDATE 1-Ruby Tuesday to explore strategic alternatives, including sale\n",
            "  - UPDATE 1-European shares inch higher after two-day sell-off\n",
            "  - UPDATE 4-Reckitt shares rise after fourth-quarter sales top estimates\n",
            "  - European shares rise, boosted by U.S. merger moves\n",
            "  - British shares dip as UK corporate woes add to trade war fears\n",
            "  - GLOBAL MARKETS-Stocks lead rally as G7 pledges virus support\n",
            "  - European shares higher on hopes of rate cut, trade optimism\n"
          ]
        }
      ],
      "source": [
        "business_recs = recommendations(business_articles_read)\n",
        "display_recommendations(business_recs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MtHUsuR8DbrX"
      },
      "source": [
        "### Query Results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eT3k7mn5DbrX"
      },
      "source": [
        "We can see that each user's recommendations have a high similarity to what the user actually reads. A user who likes Tennis news has plenty of Tennis news recommendations. A user who likes to read about Xbox has that kind of news. And a business user has plenty of Wall Street news that he or she can enjoy. \n",
        "\n",
        "Since we used only the title and the content of the article to define the embeddings, and we did not take publications and sections into account, a user may get recommendations from a publication/section that he does not regularly read. You may try adding this information when creating embeddings as well and check your query results then!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFT2bStCRB-Y"
      },
      "source": [
        "## Delete the index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ppirLic6DbrX"
      },
      "source": [
        "Delete the index once you are sure that you do not want to use it anymore. Once it is deleted, you cannot use it again.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "xcd6vDvkRB-Y"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(name=index_name)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "article_recommendations.ipynb",
      "provenance": []
    },
    "environment": {
      "name": "tf2-gpu.2-4.m61",
      "type": "gcloud",
      "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-4:m61"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.9"
    },
    "vscode": {
      "interpreter": {
        "hash": "5fe10bf018ef3e697f9035d60bf60847932a12bface18908407fd371fe880db9"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}